Gesture Theory is Linguistics: On Modelling Multimodality as Prosody

Prosody and gesture have, with few exceptions, not appealed to computational linguists, and there is even less awareness of the parallels between them, though in modelling sign languages of the hearing-impaired, the linguistic metaphor of gesture patterns as ‘phonology’ has been very fruitful. On the other hand, many disciplines from anthropological linguistics to robotics are currently occupied very productively with conversational gesture studies, but nevertheless the field is still highly fragmented. Starting from (computational) linguistic models as metaphors for the forms and functions of gestural signs, it can be shown that similarities between gesture and speech (specifically: prosody), go much further than metaphors, opening up avenues for integrating speech and gesture theory and developing a shared ontology for speech and gesture resource creation and retrieval.